Central Market
This striking building near the central train station began life as an airship hangar in the 1930s, but is now an excellent place to buy fresh produce, meat, dairy, and the occasional counterfeit goods, though these have been mostly eradicated. The market is actually quite non-touristy but very good for souvenirs. There are lots of small cafes scattered around the markets that serve up cheap local eats.
Latgalite
If you are looking for Soviet era hats, medals, pins, or complete uniforms, go to this flea market. Also has a large selection of Soviet-era surplus electronics and miscellaneous household items. If it's your dream to own an oscilloscope from the Eastern Bloc, this is not a bad place to look. On the way there you will see old buildings that aren't exactly well conserved, but there are some that are breathtaking.